外部から Cloud Logging へ fluentd で送る
といいつつ Docker
code:error
fluentd_1 | 2020-10-06 03:14:32 +0000 error: Failed to access metadata service: error_class=Net::OpenTimeout error="execution expired" fluentd_1 | 2020-10-06 03:14:32 +0000 info: Unable to determine platform fluentd_1 | 2020-10-06 03:14:32 +0000 info: using configuration file: <ROOT> fluentd_1 | <source>
...
project_id, zone, vm_id を渡せばよい
use_metadata_service false にすると最初に metadata server 叩かなくなる
こういう感じで
code:Dockerfile
FROM fluent/fluentd:v1.11-debian
USER root
RUN buildDeps="sudo make gcc g++ libc-dev" \
&& apt-get update \
&& apt-get install -y --no-install-recommends $buildDeps \
&& sudo gem install fluent-plugin-google-cloud \
&& sudo gem sources --clear-all \
&& SUDO_FORCE_REMOVE=yes \
apt-get purge -y --auto-remove \
-o APT::AutoRemove::RecommendsImportant=false \
$buildDeps \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
COPY fluent.conf /fluentd/etc/
COPY --chown=fluent:fluent credentials.json /etc/google/auth/application_default_credentials.json
USER fluent
code:fluent.conf
<source>
@type tail
path /var/log/fluent/%Y-%m-%d.log
pos_file /fluentd/log/fluent_log.pos
format json
time_key time
time_format %iso8601
tag sandbox.log
</source>
<match sandbox.log>
@type google_cloud
project_id xxxxxx
zone asia-northeast1-a
vm_id localhost
use_metadata_service false
</match>
code:docker-compose.yaml
version: '3.8'
services:
app:
build:
context: ./app
volumes:
- logs:/var/log/fluent/
fluentd:
build:
context: ./fluentd
volumes:
- logs:/var/log/fluent/
volumes:
logs:
app は適当に /var/log/fluent/yyyy-mm-dd.log にどばどば NDJSON を書く {"time": "2020-10-06T04:53:07.524Z", ... } ↑ の例は ISO8601 で時刻を書いてる
volume 共有して fluentd のコンテナと volume は共有しておく
これで実験しやすくなった
{ labels: { compute.googleapis.com/resource_name: HOSTNAME } } は勝手につく
↑ の例はコンテナ内なのでコンテナIDが HOSTNAME に入る
logName は projects/PROJECT_ID/logs/{tag}
jsonPayload.time は特殊フィールドとして扱われる(消される)
https://gyazo.com/5aeb4bda2358c24c647f356b8c22f9f2